A Memorable Marathon Match
I find myself talking about the lung cancer detector marathon match a lot. This is because people often ask me to explain what I do as a copilot of client sponsored challenges, and this is the challenge that immediately comes to mind as a perfect example. In this post I will not cover the tasks of the copilot (you can find some points here if you are interested), instead I’ll write about why I like this challenge so much.
The lung cancer detector challenge is a great marathon match for several reasons. One reason is that the goal is straightforward and easy to understand: the task is to develop an automated solution that finds and delineates tumor regions on CT scan images. It’s always good if the contest has a clear goal that can be explained in a single sentence in layman’s terms, and it is obvious to everyone that the task makes sense. Although this is the case for most of the client sponsored contests, some of them require a longer explanation and some effort of convincing the listener that running the contest is worth the effort and money.
Another reason is that the contest supports a noble cause. As the problem statement puts it: “lung cancer is the leading cause of cancer death in the United States. Many lung cancer patients receive radiation therapy, and successful treatment depends heavily on the radiation oncologist’s ability to accurately identify and delineate the tumor’s shape on medical imaging, to maximize the likelihood of killing the cancer and minimize injury to the surrounding organs. Furthermore, accurate and precise delineation of a lung tumor is important to assess changes of tumor size after treatment to understand the cancer’s responsiveness to interventions. Manual delineation of tumors is very time consuming when performed by highly trained experts and is prone to inconsistency and bias.” Of course most contestants are happy to participate in a challenge that simply improves some complex process for a client and thus makes their business more profitable, but taking part in making the world a better place is a special feeling.
A further reason, and possibly the most important one, is that the challenge was successful. It worked in the sense that it attracted a fair number of strong contestants, and also in the sense that the winning algorithms were of good quality, the stakeholders of the contest (Harvard Medical School and the Crowd Innovation Lab at Harvard) are planning to further develop the best algorithms for eventual use in real life systems.
The contest was also interesting because of the diversity of the approaches the contestants used. It was held in the beginning of 2017, around the time the “deep learning revolution” reached Topcoder. We had lots of image processing contests in earlier years which were typically won using manual feature engineering approaches. We have also held several image processing contests since this one, and with very few exceptions these are won by deep learning solutions. It seems this was the contest when these two schools of AI were in balance: three of the seven prize winning contestants (including the winner) used more traditional approaches like Random Forest classifiers, while four of them used deep learning. It was interesting – and to me, surprising – to observe how little domain knowledge the contestants built into their systems. The winning solutions know almost nothing about human anatomy, they do not try to recognize known human body parts (like the heart), they work simply by recalling image patterns seen in the training images. The only operations that resemble the way a traditional “expert system” works were trying to limit processing to the area of the lungs, trying to detect violations of symmetry in the images, and doing various minor postprocessing steps to produce more regular looking tumor regions.
One of the tasks I enjoy the most while designing a contest is to think about how to make the contestants’ lives easier, so that they can concentrate on the core task instead of having to solve mundane low level problems like interfacing with exotic, domain specific file formats. This contest was no exception. We tried to help the solvers in these two major ways: one was to convert all contest data from the field’s native DICOM format to formats that are easy to process in all languages (images in PNG and meta data in TXT), the other was to provide a visualizer and scorer tool that helped them develop their solutions off line. This is a screenshot of the visualizer tool that shows the CT scans and the ground truth tumor annotations.